System and method for three-dimensional (3D) reconstruction from ultrasound images

ABSTRACT

A method for local 3-dimensional (3D) reconstruction from 2-dimensional (2D) ultrasound images includes deriving a 2D image of an object; defining a target region within said 2D image; defining a volume scan period; during the volume scan period, deriving further 2D images of the target region and storing respective pose information for the further 2D images; and reconstructing a 3D image representation for the target region by utilizing the 2D images and the respective pose information.

Reference is hereby made to the following U.S. Provisional patentapplications whereof the benefit is hereby claimed and whereof thedisclosures are hereby incorporated by reference:

U.S. Provisional patent application No. 60/312,872, entitled MARKING 3DLOCATIONS FROM ULTRASOUND IMAGES and filed Aug. 16, 2001 in the names ofFrank Sauer, Ali Khamene, Benedicte Bascle;

U.S. Provisional patent application No. 60/312,876, entitled LOCAL 3DRECONSTRUCTION FROM ULTRASOUND IMAGES and filed Aug. 16, 2001 in thenames of Frank Sauer, Ali Khamene, Benedicte Bascle;

U.S. Provisional patent application No. 60/312,871, entitledSPATIOTEMPORAL FREEZING OF ULTRASOUND IMAGES IN AUGMENTED REALITYVISUALIZATION and filed Aug. 16, 2001 in the names of Frank Sauer, AliKhamene, Benedicte Bascle;

U.S. Provisional patent application No. 60/312,875, entitled USERINTERFACE FOR AUGMENTED AND VIRTUAL REALITY SYSTEMS and filed Aug. 16,2001 in the names of Frank Sauer, Lars Schimmang, Ali Khamene; and

U.S. Provisional patent application No. 60/312,873, entitledVIDEO-ASSISTANCE FOR ULTRASOUND GUIDED NEEDLE BIOPSY and filed Aug. 16,2001 in the names of Frank Sauer and Ali Khamene.

Reference is hereby made to the following copending U.S. patentapplications being filed on even date herewith.

U.S. patent application, entitled MARKING 3D LOCATIONS FROM ULTRASOUNDIMAGES filed in the names of Frank Sauer, Ali Khamene, Benedicte Bascle;

U.S. patent application entitled SPATIOTEMPORAL FREEZING OF ULTRASOUNDIMAGES IN AUGMENTED REALITY VISUALIZATION and filed in the names ofFrank Sauer, Ali Khamene, Benedicte Bascle;

U.S. patent application entitled USER INTERFACE FOR AUGMENTED ANDVIRTUAL REALITY SYSTEMS and filed in the names of Frank Sauer, LarsSchimmang, Ali Khamene; and

U.S. patent application entitled VIDEO-ASSISTANCE FOR ULTRASOUND GUIDEDNEEDLE BIOPSY and filed in the names of Frank Sauer and Ali Khamene.

The present invention relates to ultrasound imaging, such as for medicalimaging purposes and, more particularly to local three-dimensional (3D)reconstniction from two-dimensional (2D) ultrasound images.

Augmented Reality visualization of ultrasound images has been proposedin the literature; see for exampled, M. Bajura, H. Fuchs, and R.Ohbuchi. “Merging Virtual Objects with the Real World: Seeing UltrasoundImagery within the Patient.” Proceedings of SIGGRAPH '92 (Chicago, Ill.,Jul. 26–31, 1992). In Computer Graphics 26, #2 (July 1992): 20

Helpful background material on augmented reality and related topics canbe found in Proceedings of the IEEE and ACM International Symposium onAugmented Reality 2000, dated Oct. 5–6, 2000; Munich, Germany; IEEEComputer Society, Los Alamitos, Calif., U.S.A. In the above-citedProceedings, an article of particular interest entitled AUGMENTEDWORKSPACE: DESIGNING AN AR TESTBED is published on pages 47–53, and isauthored by Frank Sauer, an inventor in the present application, etalii.

See also the review article by R. T. Azuma: “A Survey of AugmentedReality”, Presence: Teleoperators and Virtual Environments, 6(4),355–386, (1997).

FIG. 1 show a schematic block diagram of an augmented reality system asmay be utilized in conjunction with features of the invention. A trackercamera 10 is coupled by way of and A/D (analog to digital) converter 12to a programmable digital computer 14. Two scene cameras 16 are coupledto computer 14. An ultrasound scanner 16, having a transducer 18, iscoupled by way of an A/D converter 20 to computer 14. A head-mounteddisplay (HMD) control unit 22 is coupled for signal interchange withcomputer 14 and to an HMD display 24.

Ultrasound scanners capture live 2D images from within objects orpatients (B-scans). 3D ultrasound imaging is attractive as itfacilitates the diagnosis and identification of scanned structures, andprovides overall a better understanding of the shape and topology ofsuch structures. To assemble a set of 2D images into a 3Drepresentation, one needs to track the pose of the individual 2D images,for example, using a commercial magnetic or optical tracking system.

Commercial systems are available, such as TomTec, for example, for whichsee website http://www.tomtec.de/). Free software to perform a 3Dreconstruction from 2D ultrasound slices with known poses is available;see, for example, website http://svr-www.eng.cam.ac.uk/˜rwp/stradx/.Nevertheless, it is difficult to achieve real-time performance withexisting 3D ultrasound systems.

An object of the present invention is to perform local 3D reconstructionfrom 2D ultrasound images.

In accordance with another aspect of the invention, a method for local3-dimensional (3D) reconstruction from 2-dimensional (2D) ultrasoundimages includes deriving a 2D image of an object; defining a targetregion within said 2D image; defining a volume scan period; during thevolume scan period, deriving further 2D images of the target region andstoring respective pose information for the further 2D images; andreconstructing a 3D image representation for the target region byutilizing the 2D images and the respective pose information.

The invention will be more fully understood from the following detaileddescription of preferred embodiments, in conjunction with the Drawing,in which

FIG. 1 shows a block diagram of an augmented reality system.

In accordance with an aspect of the invention, a system for local (3D)reconstruction allows a user to identify a target in an ultrasoundimage, and then to scan a volume by moving the transducer.Alternatively, the system allows a user to identify a region of interestin an ultrasound image, and then to scan a volume by moving thetransducer.

The system will perform a 3D reconstruction of the local volume that iscentered around the target or, in the alternative, bounded by the 2-Dregion of interest, and the scan motion. The local volume is smallerthan the whole volume scanned by the ultrasound slice, and the 3Dreconstruction can accordingly be performed in real time.

Preferably, the 3D information is visualized in an augmented realityfashion. The ultrasound images are overlaid onto a view of the patientor the object in a registered way. Structures visible in the ultrasoundimages appear in the location of the corresponding physical structures.Preferably, the augmented view is stereoscopic to provide the user withdepth perception. For the augmented reality visualization, the systemincludes means to track the ultrasound transducer's pose from the user'sviewpoint.

A preferred embodiment in accordance with the principles of the presentinvention comprises and ultrasound scanner, including a transducer andtracking apparatus to track the transducer. For the option withaugmented reality (AR) applications, tracking apparatus is utilized totrack user's viewpoint. A calibration procedure is established, and acomputer is utilized for generating graphics.

As concerns visualization, the volume can be shown as a 3D texture map.Alternately, the system can process the volume information and segmentout a target structure. A segmented structure is then displayed in a 3Dsurface representation.

With regard to the user interface, there are various ways for the userto input the location of the target structure of interest, includingsuch as are described in the afore-mentioned patent application entitled“MARKING 3D LOCATIONS FROM ULTRASOUND IMAGES”.

Target localization and marking can be done by using a pointing device,such as a computer-type mouse to mark a 2D location in the image. Withthe knowledge of the 3D pose of that image, the system can calculate the3D position of the marker. The user's input may be in an “on-line” mode,where he holds the transducer still in the desired pose, or in an“off-line” mode, where he first freezes (that is, records) the imagetogether with its pose information, and then places the marker in therecorded still image at his leisure. For the on-line mode, the pointingdevice is preferably attached to the transducer and can be operated withthe same hand that is holding the transducer, or alternatively, isplaced on the floor and is foot-actuated by the user.

Preferably, the system performs image processing on an ultrasound imageand automatically or semiautomatically locates a target. The type oftarget may have optionally been predetermined by the user. Hence, theinput of the user is simplified. For example, without an extra pointingdevice, the user may place the transducer in a pose where the targetstructure appears on the vertical center-line of the ultrasound image.The user then simply triggers the system to locate the target along theimage's center-line—with a button on the transducer, with a foot switch,or by voice control, and so forth.

The system includes a processor that searches the image along itscenterline, which makes locating the target easier than if the searchwould have had to be conducted over the whole image. A preferred searchalgorithm would be to de-noise the image around the centerline, e.g.with a median filter, identify potential target locations in a line scanalong the centerline, and verify the existence of a target with a Houghtransform. The Hough transform is known and may be found in varioustextbooks, such as, for example, “Fundamentals of Electronic ImageProcessing” by Arthur R. Weeks, Jr., IEEE Press, New York, N.Y.; 1996.When the system proposes a target location, the user then accepts orrejects it with another simple trigger input (such as button ontransducer, footswitch, voice, and so forth).

Alternatively, a line on the ultrasound image may be utilized aspointer. A line other than a vertical centerline can be used, such as avertical off-center line, or a line that is tilted at an angle withrespect to vertical direction. Also, target position along line can beinput by user via thumbwheel at the transducer or by using a similar 1-Dpointing device, (no image processing being necessary.

Alternatively the user can use a line on the ultrasound image to pointto a target from two different transducer poses, so that the system cancalculate the target location as the intersection of the two lines in 3Dspace; two different lines can be used to require less movement betweenthe two pointing transducer poses, e.g. two lines that intersect in theimage.

The image processing is powerful enough to find a target in the 2Dimage. Then the user need only position the transducer so that thetarget is visible in the image, initiate the automatic target search,and then confirm or dismiss a target proposed by the system.

Several targets can be found in the same image. This embodiment isattractive from a user point of view as it requires the least input, butrobust target detection in ultrasound images is generally verydifficult. Therefore, it is preferable to provide the user with a 1D or2D pointing interface as described in the above.

The size of a region of interest centered at the target can be fixed,but is preferably user-adjustable. Once size and location of a region ofinterest is determined in a 2D ultrasound slice, the user triggers thestart and the end of the volume scan, e.g. by using a pushbutton at thetransducer, or with a foot switch, or via voice command, etc. The systemrecords 2D images together with the pose information from the trackingsystem and builds the volume according to prior art.

In an alternate embodiment, the system automatically selects targetstructures for local reconstruction. An important example is local 3Dreconstruction from ultrasound Doppler images. The system automaticallydetects the regions with flow and performs local volume reconstructionor segmentation for these regions.

While the invention has been explained by way of exemplary embodiments,it will be understood by one of skill in the art to which it pertainsthat various modifications and changes may be readily made without(departing from the spirit of the invention which is defined by theclaims following.

1. A method for local 3-dimensional (3D) reconstruction from2-dimensional (2D) ultrasound images, comprising: deriving a 2D image ofan object, together with corresponding pose information; defining a 2Dtarget region within said 2D image, the 2D target region having auser-defined size and location; defining further a local target volumewithin the object, the target volume having a user-defined size andlocation; deriving a set of further 2D images having respective posesand intersecting said local target volume, said set of further 2D imagesscanning through the local target volume; and reconstructing a 3D imagerepresentation of said local target volume from said set of 2D imagesand said respective poses.
 2. A method for local 3D reconstruction asrecited in claim 1, wherein said step of defining a target regioncomprises a step of searching said image along its centerline foridentifying a potential target region.
 3. A method for local 3Dreconstruction as recited in claim 2, wherein said step of searchingsaid image comprises a step of utilizing a search algorithm forsearching said image along its centerline for identifying a potentialtarget region.
 4. A method for local 3D reconstruction as recited inclaim 3, wherein said step of utilizing a search algorithm comprises astep of do-noising said image around its centerline for identifying apotential target region.
 5. A method for local 3D reconstruction asrecited in claim 4, wherein said step of de-noising comprises a step ofmedian filtering for identifying a potential target region.
 6. A methodfor local 3D reconstruction as recited in claim 3, wherein said step ofsearching said image comprises a step of utilizing a Hough transform forverifying a potential target region.
 7. The method of claim 1 whereinthe step of defining a 2D target region further comprises the step of:marking a 2D location in the image.
 8. The method of claim 1 wherein the2D target region is rectangular.
 9. The method of claim 1 wherein thelateral extent of said 3D target volume is determined by the 2D targetregion, and the depth of said 3D target volume is determined involvingthe steps of moving the ultrasound transducer used for said deriving ofsaid 2D images into a start position, moving said ultrasound transducerinto an end position, wherein the movement is essentially perpendicularto the 2D image planes.
 10. The method of claim 1 wherein the step ofdefining a 2D target region further comprises the steps of: predefininga target to be found in the image; automatically searching for targetsin the image; and if a target is found, marking the target.
 11. A methodfor local 3D reconstruction as recited in claim 1, wherein said step ofdefining a 2D target region includes identifying a target in said 2Dimage.
 12. A method for local 3-dimensional (3D) reconstruction from2-dimensional (2D) ultrasound images, comprising: deriving a 2D image ofan object; defining a 2D target region within said 2D image, the 2Dtarget region having a user-defined size and location; defining furthera local target volume within the object, the target volume having auser-defined size and location; defining a volume scan period; duringsaid volume scan period, deriving a set of further 2D images of saidtarget region that intersect said local target volume, said set offurther 2D images scanning through the local target volume, and storingrespective pose information for said set of further 2D images; andreconstructing a 3D image representation of said local target volume byutilizing said set of 2D images and said respective pose information.13. A method for local 3D reconstruction as recited in claim 12, whereinsaid steps of defining a target region comprises semi-automatic steps.14. A method for local 3-dimensional (3D) reconstruction from2-dimensional (2D) ultrasound images, comprising: deriving a 2D image ofan object; defining a 2D target region within said 2D image, said 2Dtarget region being significantly smaller than the 2D image and having auser-defined size and location; defining further a local target volumewithin the object, the target volume having a user-defined size andlocation; defining the start and end of a volume scan; deriving a set offurther 2D images of said target region during a period between saidstart and end of said volume scan, said set of further 2D imagesintersecting said local target volume and scanning through the localtarget volume, said set of further 2D images having respective poses;and reconstructing a 3D image representation fey of said local targetvolume by utilizing said set of 2D images and said respective poseinformation.
 15. A method for local 3-dimensional (3D) reconstructionfrom 2-dimensional (2D ) ultrasound Doppler images, comprising: derivinga 2D Doppler image of an object; detecting flow regions exhibitingpredetermined flow characteristics; defining a 2D target region withinsaid 2D image in correspondence with said flow regions, the 2D targetregion having a user-defined size and location; defining further a localtarget volume within the object, the target volume having a user-definedsize and location; defining a volume scan period; during said volumescan period, deriving a set of further 2D images of said 2D targetregion and storing respective pose information for said set of further2D images, said set of further 2D images intersecting said local targetvolume and scanning through said local target volume; and reconstructinga 3D image representation of said local target volume by utilizing saidset of 2D images and said respective pose information.
 16. Apparatus forlocal 3-dimensional (3D) reconstruction from 2-dimensional (2D)ultrasound images, comprising: means for deriving a 2D image of anobject; means for defining a 2D target region within said 2D image, the2D target region having a user-defined size and location; means fordefining further a local target volume within said object, the targetvolume having a user-defined size and location; means for defining avolume scan period; means for defining size and location of 2D targetregion; means for defining size and location of 3D target volume; meansfor storing respective pose information for a set of further 2D imagesof said target region derived during said volume scan period by saidmeans for deriving a 2D image said set of further 2D images intersectingsaid local target volume and scanning through the local target volume;and means for reconstructing a 3D image representation of said localtarget volume by utilizing said set of 2D images and said respectivepose information.
 17. Apparatus for local 3-dimensional reconstructionas recited in claim 16, wherein said means for defining a target regioncomprises processor means for searching said image along its centerlinefor identifying a potential target region.
 18. Apparatus for local3-dimensional (3D) reconstruction as recited in claim 17, whereinprocessor means for searching utilizes a search algorithm for searchingsaid image along its centerline for identifying a potential targetregion.
 19. Apparatus for local 3-dimensional (3D) reconstruction asrecited in claim 17, wherein processor means for searching utilizes asearch algorithm for de-noising said image around its centerline foridentifying a potential target region.
 20. Apparatus for local3-dimensional (3D) reconstruction as recited in claim 17, whereinprocessor means for searching utilizes a search algorithm for de-noisingsaid image around its centerline, by using a median filter, foridentifying a potential target region.
 21. Apparatus for local3-dimensional (3D) reconstruction as recited in claim 17, whereinprocessor means for searching utilizes a Hough transform for verifying apotential target region.
 22. The method of claim 7 wherein a pointingdevice is used to mark the location in the image.
 23. The method ofclaim 7 wherein said marked 2D location is used in conjunction with saidcorresponding pose information to mark a corresponding 3D location. 24.The method of claim 7 wherein a user can adjust the size of the 2Dtarget region.
 25. The method of claim 7 wherein the 2D target region iscentered at said marked 2D location.
 26. The method of claim 22 whereinthe pointing device is a computer mouse.
 27. The method of claim 22where the pointing device is the tracked head movement of the user. 28.The method of claim 8 wherein the step of defining a 2D target regionfurther comprises the step of: determining width and height of said 2Dtarget region.
 29. The method of claim 8 wherein said 3D target volumehas a rectangular cross-section, and height and width of said 3D targetvolume are identical to the height and width of said 2D target region.30. The method of claim 28 wherein width and/or height are predefined.31. The method of claim 28 wherein the user sets width and/or heightaccording to the size of the target.
 32. The method of claim 28 whereinan automatic algorithm sets width and/or height according to the size ofthe target.
 33. The method of claim 29 wherein the depth of said 3Dtarget volume is pro-set or pro-selected.
 34. The method of claim 29wherein the depth of said 3D target volume is set proportional to itsheight or width.
 35. The method of claim 29 wherein the user sets thedepth of said 3D target volume according to the size of the target. 36.The method of claim 29 wherein an algorithm sets the depth of said 3Dtarget volume according to the size of the target.
 37. The method ofclaim 23 wherein said 3D target volume is centered at said marked 3Dlocation.
 38. The method of claim 9 wherein a trigger device is used toidentify said start and end positions.
 39. The method of claim 38wherein said trigger device is a computer mouse.
 40. The method of claim38 wherein said trigger device is a foot switch.
 41. The method of claim38 wherein said trigger device is located at the transducer.
 42. Themethod of claim 10 wherein a user can adjust the size of the 2D targetregion.
 43. The method of claim 42 further comprising the steps of:triggering a start and end for a volume scan of the target; building a3D target volume from the volume scan.